ABSTRACT
International research programs for students offer an important opportunity to support students in developing skills in both research and intercultural competence. During the COVID-19 pandemic, many of these programs made the shift to operating virtually, with likely impacts on program outcomes. The purpose of this study was to identify the approaches that program leaders used in adapting international research programs to the virtual environment and explore how these innovations could inform the design of these programs going forward. We conducted eight focus groups with over 40 U.S.-based faculty who had experience running these programs to understand the benefits, challenges, and future potential of incorporating virtual elements into international research programs for students. This paper reports the results of these focus groups and provides suggestions for future program design based on best practices and innovations identified through the development of virtual programs. © 2022 IEEE.
ABSTRACT
This research paper compares the results of a novel computer-assisted approach for analyzing a large volume of open-ended responses with those of a more traditional open coding approach. The work is motivated by the observation that in engineering education ecosystems, community members produce text through myriad activities both inside and outside of the classroom in teaching and research settings. In many of these cases, there is an abundance of text available to educators and researchers that could provide insight into various phenomena of interest within the system - student conceptual understanding, student experiences outside the classroom, how instructors can improve their teaching, or even shifts in collective conversations. Unfortunately, while these bodies of text have the potential to provide novel insights to educators and researchers, traditional analysis techniques do not scale well. For example, analyzing larger amounts of text can take one grader or researcher significantly more time than grading a small set of text responses. A larger body of text also creates more challenges for intrarater reliability. Likewise, expanding the size of the grading or research team can create interrater reliability challenges and the possibility of bias. To address this opportunity, we have created a natural language processing system that augments human analysis so as to facilitate and enhance the work of one person (or team). Specifically, we take minimally pre-processed text, embed them using a pre-trained transformer (a specific kind of neural network architecture trained to encode inputs and decode outputs), and perform a sequence of dimension reduction techniques capped with a final clustering step. Such a system can help reduce the amount of time needed to analyze the text by effectively running a first pass on the text to group similar responses together. The human user can utilize these groupings to perform further analysis to fine tune and identify meanings in ways that only a human could. The system also can help improve consistency by analyzing across the entire collection of texts simultaneously and grouping similar items together. This is in contrast with a single person or a team that would have to work in series, analyzing responses sequentially and thereby creating the potential for inconsistencies across time. In this paper we describe the system's architecture and data processing steps. We demonstrate the utility of this approach by applying the method on three questions from an end-of-semester feedback survey in a large, required introductory engineering course. The survey questions were part of a general feedback survey and asked students about their experiences in the transition to online learning subsequent to the SARS-CoV-2 outbreak.. Our results suggest that the pre-analysis text clustering can improve speed and accuracy of coding when compared with unassisted human coding-the system augments what we have traditionally done in coding, grading, or making sense of large quantities of textual data. As natural language processing techniques continue to develop, the engineering education research community should continue to explore potential applications to improve understanding and sensemaking from large volumes of underutilized text data from both within and outside of classroom settings. © American Society for Engineering Education, 2021